OpenAI and Anthropic are ignoring an established rule that prevents bots scraping online content

• Generative AI tools are based on models that use huge amounts of content scraped from the web.
• OpenAI and Anthropic have said publicly they respect robots.txt and blocks to their web crawlers.
• Yet, both companies are ignoring or circumventing such blocks, BI has learned.

The world’s top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.